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A central challenge for cognitive and systems neuroscience is to relate the incredibly complex behavior 
of animals to the equally complex activity of their brains. Recently described, large-scale neural models 
have not bridged this gap between neural activity and biological function. In this work, we present a 
2.5-million-neuron model of the brain (called “Spaun”) that bridges this gap by exhibiting many different 
behaviors. The model is presented only with visual image sequences, and it draws all of its responses with 
a physically modeled arm. Although simplified, the model captures many aspects of neuroanatomy, 
neurophysiology, and psychological behavior, which we demonstrate via eight diverse tasks. 


arge-scale neural simulations are becom- 
| ing increasingly common [see (7) for a 
review]. These include the ambitious Blue 
Brain Project (2), which has simulated about 
1 million neurons in cortical columns and includes 
considerable biological detail, accurately reflect- 
ing spatial structure, connectivity statistics, and 
other neural properties. More recent work has sim- 
ulated many more neurons, such as the 1 billion 
neurons simulated in the Cognitive Computa- 
tion Project (3), which has been hailed as a cat- 
scale simulation. A human-scale simulation of 
100 billion neurons has also been reported (4). 

Although impressive scaling has been achieved, 
no previous large-scale spiking neuron models have 
demonstrated how such simulations connect to a 
variety of specific observable behaviors. The focus 
of this past work has been on scaling to larger num- 
bers of neurons and more detailed neuron models. 
Unfortunately, simulating a complex brain alone 
does not address one of the central challenges for 
neuroscience: explaining how complex brain activity 
generates complex behavior. In contrast, we present 
here a spiking neuron model of 2.5 million neurons 
that is centrally directed to bridging the brain- 
behavior gap. Our model embodies neuroanatomical 
and neurophysiological constraints, making it di- 
rectly comparable to neural data at many levels of 
analysis. Critically, the model can perform a wide 
variety of behaviorally relevant functions. We show 
results on eight different tasks that are performed 
by the same model, without modification. 

All inputs to the model are 28 by 28 images of 
handwritten or typed characters. All outputs are 
the movements of a physically modeled arm 
that has mass, length, inertia, etc. For convenience, 
we refer to the model as “Spaun” (Semantic Point- 
er Architecture Unified Network) (see Fig. 1 and 
supplementary materials and methods section S1.1). 
Many of the tasks we have chosen are the subject of 
extensive modeling in their own right [e.g., image 
recognition (5, 6), serial working memory (WM) 
(7, 8), and reinforcement learning (RL) (9, 10)], 
and others demonstrate abilities that are rare for 
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neural network research and have not yet been dem- 
onstrated in spiking networks (e.g., counting, question 
answering, rapid variable creation, and fluid reason- 
ing). The eight tasks (termed “A0” to “A7”) that 
Spaun performs are: (A0) Copy drawing. Given a 
randomly chosen handwritten digit, Spaun should 
produce the same digit written in the same style 
as the handwriting (movie S1; all supplemental 
movies can be viewed at http://nengo.ca/build-a- 
brain/spaunvideos). (A1) Image recognition. Given 
arandomly chosen handwritten digit, Spaun should 
produce the same digit written in its default writ- 
ing (movie S2). (A2) RL. Spaun should perform 
a three-armed bandit task, in which it must deter- 
mine which of three possible choices generates the 
greatest stochastically generated reward. Reward 
contingencies can change from trial to trial (movie 
S3). (A3) Serial WM. Given a list of any length, 
Spaun should reproduce it (movie S4). (A4) Count- 
ing. Given a starting value and a count value, Spaun 
should write the final value (that is, the sum of the 
two values) (movie S5). (A5) Question answering. 
Given a list of numbers, Spaun should answer 
either one of two possible questions: (i) what is in 
a given position in the list? (a “P” question) or (ii) 
given a kind of number, at what position is this 
number in the list? (a “K” question) (movie S6). 
(A6) Rapid variable creation. Given example syn- 
tactic input/output patterns (e.g.,0 0 74 — 7 4; 
0024-24: etc.), Spaun should complete a novel 
pattern given only the input (e.g., 0 0 1 4 — ?) 
(movie S7). (A7) Fluid reasoning. Spaun should 
perform a syntactic or semantic reasoning task 
that is isomorphic to the induction problems from 
the Raven’s Progressive Matrices (RPM) test for 
fluid intelligence (//). This task requires completing 
patterns of the form: 1 2 3; 5 6 7; 3 4 ? (movie S8). 
Each input image is shown for 150 ms and separated 
by a 150-ms blank (see table S2 for example inputs 
for each task). The model is told what the task will 
be by showing it an “A” and the number of the task 
(0 to 7). The model is then shown input defining 
the task (see Figs. 2 and 3 for examples). Spaun is 
robust to invalid input (fig. S10) and performs 
tasks in any order without modeler intervention. 

Figure 1A shows the anatomical architecture 
of the model. Connectivity and functional ascrip- 
tions to brain areas in Spaun are consistent with 
current empirical evidence (table S1). In general, 


we modeled neuron and synaptic response prop- 
erties on the electrophysiology literature for the 
relevant anatomical areas. For instance, the basal 
ganglia have largely GABAergic neurons, with 
dopamine modulating learning in the striatum, 
and the cortex has largely N-methyl-p-aspartate and 
AMPA synaptic connections (supplementary sec- 
tion S1.3). As a result, the dynamics in the model 
are tightly constrained by underlying neural prop- 
erties (see supplementary section $2.4). 

The functional architecture of the model is de- 
scribed in Fig. 1B. The network implementing the 
Spaun model consists of three compression hierar- 
chies, an action-selection mechanism, and five sub- 
systems. Components of the model communicate 
using spiking neurons that implement neural rep- 
resentations that we call “semantic pointers,” using 
various firing patterns. Semantic pointers can be 
understood as being elements of a compressed 
neural vector space (supplementary sections S1.1 
and $1.2). Compression is a natural way to under- 
stand much of neural processing. For instance, the 
number of cells in the visual hierarchy gradually 
decreases from the primary visual cortex (V1) to the 
inferior temporal cortex (IT) (/2), meaning that the 
information has been compressed from a higher- 
dimensional (image-based) space into a lower- 
dimensional (feature) space (supplementary section 
$1.3). This same kind of operation maps well to the 
motor hierarchy (/3), where lower-dimensional firing 
patterns are successively decompressed (for ex- 
ample, when a lower-dimensional motor represen- 
tation in Euclidean space moves down the motor 
hierarchy to higher-dimensional muscle space). 

Compression is functionally important because 
low-dimensional representations can be more ef- 
ficiently manipulated for a variety of neural compu- 
tations. Consequently, learning or defining different 
compression/decompression operations provides a 
means of generating neural representations that 
are well suited to a variety of neural computations. 
The specific compression hierarchies in Spaun are 
(see Fig. 1B): (i) a visual hierarchy, which com- 
presses image input into lower-dimensional firing 
patterns; (ii) a motor hierarchy that decompresses 
firing patterns in a low-dimensional space to drive 
a simulated arm; and (iii) a WM, which constructs 
compressed firing patterns to store serial position 
information. The WM subsystem includes several 
subcomponents that provide stable representations 
of intermediate task states, task subgoals, and context. 

Spaun’s action-selection mechanism is based 
on a spiking basal ganglia model that we have 
developed in other work (/4) but is here extended 
to process higher-dimensional neural represen- 
tations. The basal ganglia determine which state 
the network should be in, switching as appro- 
priate for the current task goals. Consequently, 
the model’s functional states are not hardwired, 
as the basal ganglia are able to control the order 
of operations by changing information flow be- 
tween subsystems of the architecture. 

The five subsystems, from left to right in Fig. 
1B, are used to: (i) map the visual hierarchy firing 
pattern to a conceptual firing pattern as needed 
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(information encoding), (11) extract relations between 
input elements (transformation calculation), (iii) 
evaluate the reward associated with the input (re- 
ward evaluation), (iv) decompress firing patterns 
from memory to conceptual firing pattern (informa- 
tion decoding), and (v) map conceptual firing pat- 
terns to motor firing patterns and control motor 
timing (motor processing). Supplementary materials 
section S1.3 includes a more detailed description 
of each element. It is critical to note that the ele- 
ments of Spaun are not task-specific. That is, they 
are used in a variety of combinations to perform the 
chosen tasks, resulting in the same circuitry being 
used across tasks. This makes it straightforward 
to extend the model to some new tasks (supple- 
mentary section S2.4). 
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The neural connection weights of these sub- 
systems can be learned with a biologically plau- 
sible spike-based rule (75), although we use more 
efficient optimization methods to determine the 
synaptic weights (supplementary section S1.2). 

To help explain the functioning of the model, 
we consider the serial WM task. Figure 2A shows 
the information flow through the model for this 
task. The storage and recall states of the network 
are common to many tasks. For the WM task, 
these states occur immediately one after the other, 
although the delay is task-dependent. Initially, see- 
ing the task identifier (A3) switches Spaun into the 
storage state. In the storage state, the network com- 
presses the incoming image into a visually based 
firing pattern (FP in the figure) that encodes vi- 
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Fig. 1. Anatomical and functional architecture of Spaun. (A) The anatomical architecture of Spaun shows the 
major brain structures included in the model and their connectivity. Lines terminating in circles indicate 
GABAergic connections. Lines terminating in open squares indicate modulatory dopaminergic connections. Box 
styles and colors indicate the relationship with the functional architecture in (B). PPC, posterior parietal cortex; 
M1, primary motor cortex; SMA, supplementary motor area; PM, premotor cortex; VLPFC, ventrolateral pre- 
frontal cortex; OFC, orbitofrontal cortex; AIT, anterior inferior temporal cortex; Str, striatum; vStr, ventral 
striatum; STN, subthalamic nucleus; GPe, globus pallidus externus; GPi, globus pallidus internus; SNr, sub- 
stantia nigra pars reticulata; SNc, substantia nigra pars compacta; VTA, ventral tegmental area; V2, secondary 
visual cortex; V4, extrastriate visual cortex. (B) The functional architecture of Spaun. Thick black lines indicate 
communication between elements of the cortex; thin lines indicate communication between the action- 
selection mechanism (basal ganglia) and the cortex. Boxes with rounded edges indicate that the action- 
selection mechanism can use activity changes to manipulate the flow of information into a subsystem. The 
open-square end of the line connecting reward evaluation and action selection denotes that this connection 
modulates connection weights. See table S1 for more detailed definitions of abbreviations, a summary of the 
function to anatomy mapping, and references supporting Spaun’s anatomical and functional assumptions. 
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sual features, maps that firing pattern to another 
firing pattern that represents the related concept 
(e.g., “TWO”; see supplementary section $1.3), and 
then compresses that firing pattern into a memory 
trace that is stored in WM. The compression op- 
erator (i.e., “&”) binds the concept firing pattern 
(e.g., TWO) to a position representation (e.g., P3) 
and adds the result (i.e., TWO & P3, as in Fig. 
2C) to WM. As shown in Fig. 2C, this process is 
repeated as long as items are shown to the model. 

Figure 2B shows a screen capture from a 
movie of the WM simulation. When the mod- 
el sees the “?” input (as in Fig. 2B), the basal 
ganglia reroute cortical connectivity to allow 
Spaun to recall the input stored in the dorsolateral 
prefrontal cortex (DLPFC). Recall consists of de- 
compressing an item from the stored representation 
of the full list, mapping the resulting concept 
vector to a known high-level motor command, 
and then decompressing that motor command 
to specific joint torques to move the arm. This 
process is repeated for each position in the WM, 
to generate Spaun’s full written response. Figure 
2C shows the entire process unfolding over time, 
including spike rasters, conceptual decodings of 
the contents of DLPFC, and the input and output. 

Critically, no single task captures the distinct 
features of this model. To highlight the diver- 
sity of tasks the model is able to perform, Fig. 3 
shows the results of the model performing a low- 
level perceptual-motor task (the copy-drawing 
task), as well as a challenging pattern-induction 
task only performed by humans (the RPM task). 

Specifically, Fig. 3A demonstrates that the low- 
level perceptual features in the input are available 
to Spaun to drive its motor behavior. Figure 3B 
demonstrates the RPM task for one sample pattern 
(see fig. S6 for an additional example). In this 
task, Spaun is presented with two groups of three 
related items and must learn the relation between 
items in the groups. Spaun then uses its inferred 
relation to complete the pattern of a third set of 
items. Similarity plots for the DLPFC show con- 
ceptual decodings of neural activities. The model 
learns the relation between subsequent strings of 
numbers by comparing patterns in DLPFC1 and 
DLPFC2 (see supplementary section $1.3). Hu- 
man participants average 89% correct (chance is 
13%) on the matrices that include only an induc- 
tion rule (5 of 36 matrices) (/6). Spaun performs 
similarly, achieving a match-adjusted success rate 
of 88% (see supplementary section $2.3). 

To demonstrate that Spaun captures general 
psychological features of behavior, it is critical to be 
able to simulate populations of participants. Every 
time a specific instance of Spaun is generated, the 
parameters of the neurons are picked from random 
distributions (supplementary section $1.4). Conse- 
quently, generating many instances allows for compar- 
ison with population-wide behavioral data. Figure 4 
compares the recall accuracy of the model as a func- 
tion of list length and position in a serial recall task 
to human population data. As with human data (17), 
Spaun produces distinct recency (items at the end 
are recalled with greater accuracy) and primacy 
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(items at the beginning are recalled with greater 
accuracy) effects. A good match to human data from 
a rapid serial-memory task using digits and short 
presentation times (78) is also evident, with 17 of 22 
human mean values within the 95% confidence in- 
terval of 40 instances of the model. Additional 
population comparisons are presented in fig. S8. 
To this point, we have only described performance 
on three of the eight tasks that Spaun performs. The 
tasks not yet discussed are: (1) image recognition, 
for which the model achieves 94% accuracy on un- 
trained data from the MNIST handwriting database 
[human accuracy is ~98% (19)]; (ii) RL, for which 
the model is able to learn reward-dependent actions 
in a variable environment using known neural mech- 
anisms (fig. S5); (iii) counting, for which the model 
reproduces human reaction times and scaling of var- 
iability (fig. S8A); (iv) question answering, for 
which the model generates a novel behavioral pre- 
diction (fig. S8B); and (v) rapid variable creation, 
for which the model instantiates the first neural ar- 
chitecture able to solve this challenging task (fig. S11). 
However, the central purpose of this work is 
not to explain any one of these tasks, but to propose 
a unified set of neural mechanisms able to perform 
them all. In a sense, the complex task solved by 
Spaun is one of coordination. That is, the rapid 
flexibility of biological systems is its target of ex- 
planation. The specific dynamics of Spaun’s responses 
to this wide variety of tasks is governed by four 
parameters, each of which is set empirically (the 
time constants of the neurotransmitters; see sup- 
plementary section $2.4). Thus, without fitting, the 
model is consistent with dynamics from single cells 
and behavior (see figs. S8 and S11 and supplemen- 
tary section $2.4), and is able to switch between a 
wide variety tasks quickly and robustly (fig. S10). 
Although Spaun’s main contribution lies in its 
breadth, it also embodies new hypotheses regard- 
ing how specific tasks are solved. For instance, 
the proposed method of solving the rapid variable- 
creation task is distinct to Spaun [this task has 
been identified as one that no contemporary neural 
architecture could perform as quickly as humans 
(20)], as is the account of serial WM. Such hy- 
potheses have resulted in new testable predictions 
(figs. S8 and S9). Still, Spaun’s uniqueness lies 
in its being a platform for exploring the robust 
flexibility of biological cognition. Consider the ex- 
ample of learning: Learning in Spaun takes on 
many forms. Although learning takes place in the 
RPM, rapid variable-creation, and RL, connection- 
weight changes only occur in the RL task (supple- 
mentary section $2.4). In most neural models, this 
kind of learning is often used as the main method 
of model construction, and it is possible to learn all 
of the elements of the Spaun model in this tra- 
ditional sense (supplementary section $2.4). How- 
ever, constructing models in this manner does not 
address a central, difficult challenge of learning in 
biological brains. That challenge consists of explain- 
ing how robust learning can occur in a continuous- 
ly operating, complex, and multifunctional brain. 
Spaun minimally demonstrates this kind of 
learning in the RL task, as connection-weight 
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Fig. 2. The serial WM task. (A) Information flow through Spaun during the WM task. Line style and 
color indicate the element of the functional architecture in Fig. 1B responsible for that function. 
FP, firing pattern. (B) A screen capture from the simulation movie of this task (supplementary 
section $2.1), taken at the 2.5-s mark of the time course plot in (C). The input image is on the 
right, the output is drawn on the surface below the arm. Spatially organized (neurons with similar 
tuning are near one another), low-pass—filtered neuron activity is approximately mapped to 
relevant cortical areas and shown in color (red is high activity, blue is low). Thought bubbles show 
spike trains, and the results of decoding those spikes are in the overlaid text. For Str, the thought 
bubble shows decoded utilities of possible actions, and in GPi the selected action is darkest. (C) 
Time course of a single run of the serial WM task. The stimulus row shows input images. The arm 
row shows digits drawn by Spaun. Other rows are labeled by their anatomical area. Similarity plots 
(solid colored lines) show the dot product (i.e., similarity) between the decoded representation 
from the spike raster plot and concepts in Spaun’s vocabulary. These plots provide a conceptual 
decoding of the spiking activity, but this decoding is not used by the model (supplementary section 
$1.1). Raster plots in this figure are generated by randomly selecting 2000 neurons from the 
relevant population and discarding any neurons with a variance of less than 10% over the run. 
® denotes the convolution compression operator. 
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changes do not adversely affect the performance 
of the model on other tasks. Smaller-scale models 
cannot provide even this minimal demonstration, 
because they lack the variety of tasks necessary 
to demonstrate robustness. As such, Spaun pro- 
vides a distinct opportunity to test learning algo- 
rithms in a challenging but biologically plausible 
setting. More generally, Spaun provides an op- 
portunity to test any neural theory that may be 
affected by being embedded in a complex, dynam- 
ical context, reminiscent of a real neural system. 


A 


However, Spaun has little to say about how 
that complex, dynamical system develops from 
birth. Furthermore, Spaun has many other limi- 
tations that distinguish it from developed brains. 
For one, Spaun is not as adaptive as a real brain, 
as the model is unable to learn completely new 
tasks. In addition, both attention and eye position 
of the model is fixed, making Spaun unable to 
control its own input. Also, its perceptual and con- 
ceptual representations are largely limited to the 
space of digits from 0 to 9. Anatomically, many 
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Fig. 3. Time-course plots for two Spaun tasks. (A) Results of the copy-drawing task. The input/output 
pairs for 20 additional runs are shown to the right. (B) Results of an example run of the RPM task, 
plotted using the same method as described in Fig. 2C. See text for details. 


A Data 


Accuracy 
oO 
N 


0.6 
= -4 items 
0.5 — 5 items 
< = -6 items 
0.1 7 items 
0 


1 2 3 4 5 6 7 
Item Position 


B Model 


Accuracy 
o o 
N œ 


° 
© 


Item Position 


Fig. 4. Population-level behavioral data for the WM task. Accuracy is shown as a function of position and list 
length for the serial WM task. Error bars are 95% confidence intervals over 40 runs per list length. (A) Human 
data taken from (18) (only means were reported). (B) Model data showing similar primacy and recency effects. 
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areas of the brain are missing from the model. 
Those that are included have too few neurons and 
perform only a subset of functions found in their 
respective areas. Physiologically, the variability of 
spiking in the model is not always reflective of 
the variability observed in real brains (table S3). 
However, we believe that, as available computa- 
tional power increases, many of these limitations can 
be overcome via the same methods as those used 
to construct Spaun (supplementary section $2.4). 
Even in its current form, Spaun offers a dis- 
tinctly functional view and set of hypotheses 
regarding the neural mechanisms and organi- 
zation that may underlie basic cognitive functions. 
Consequently, Spaun opens new avenues for test- 
ing ideas about biological cognition under bi- 
ologically plausible, more complex, and more 
functional settings than previously available. 
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